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EXECUTIVE  SUMMARY 


INTRODUCTION 

Developing  a  system  that  promotes  ease  of  interaction  with  the  computer  can  not  only  aid  learners 
in  the  effective  assimilation  of  new  or  difficult  material,  but  may  also  inhibit  any  demotivating  influences 
which  may  result  from  difficulty  in  interacting  with  the  system.  Incorporating  good  user-computer  interface 
(UCI)  in  a  computer-aided  instruction  (CAI)  exercise  that  is  designed  in  accordance  with  human  factors 
guidelines  can  lead  to  increased  training  efficiency  by  reducing  stress  and  errors  made  on  the  part  of  the 
learner. 

Hamel  and  Clark  (1)  developed  a  checklist  based  on  five  human  factors  principles  found  to 
contribute  to  good  UCI  as  outlined  by  Williges  and  Williges  (2).  These  five  principles  (brevity, 
consistency,  flexibility,  compatibility,  and  responsiveness)  are  made  up  of  identifying  characteristics  that 
were  embedded  into  an  experimental  training  program,  as  they  have  never  been  formally  studied,  nor 
empirically  assessed  as  to  their  contribution  to  UCI. 

The  objective  of  this  research  was  to  evaluate  how  subjects  rated  these  characteristics'  contribution 
towards  ease  of  interaction  and  training  gain  and  to  develop  a  handbook  for  CAI  designers  with  a  weighted 
scoring  technique  based  on  these  subject  ratings. 

METHOD 

Forty-five  subjects  from  the  University  of  Central  Florida  voluntarily  participated  in  this 
experiment  An  IBM  PC/XT  with  single  floppy,  hard  disk  drive,  640  KB  memory,  and  color  monitor  was 
used  to  run  the  pretest,  the  CAI  program  embedding  the  UCI  characteristics,  and  the  posttest,  accordingly, 
for  each  subject,  on  an  individual  basis.  A  questionnaire  was  developed  using  a  seven-point  Liken  scale  so 
that  subjects  could  rate  the  perceived  effectiveness  of  the  UCI  characteristics  with  regards  to  ease  of 
interaction  and  training  gain.  Additionally,  the  questionnaire  was  converted  to  a  handbook  with  illustrative 
examples  for  courseware  designers  to  use  in  evaluating  the  user-computer  interface  of  a  CAI  exercise. 

RESULTS  AND  DISCUSSION 


The  data  recorded  was  subjected  to  a  full  model  multiple  regression  correlation  analysis  to 
determine  the  contribution  of  each  category  to  ease  of  use  of  the  CAI  program.  The  results  of  the  analyses 
indicated  that  each  category  significantly  contributed  to  perceived  ease  of  use  of  the  CAI  exercise.  This 
empirically  substantiates  the  principles  proposed  by  Williges  and  Williges  (2).  Subjective  ratings  of  the 
importance  of  each  characteristic  to  ease  of  use  and  training  gain  were  averaged  for  each  of  the  five 
categories.  Category  means  were  used  in  a  multiple  regression  analysis  to  predict  actual  training  gain. 
Responsiveness  contributed  a  significant  amount  of  the  variance  in  both  sets  of  ratings.  Consistency  made 
a  significant  contribution  to  the  variance  when  the  ease  of  use  ratings  were  analyzed. 

The  results  provide  a  checklist  with  a  weighted  scoring  method  for  the  evaluation  of  CAI.  The 
data  also  provides  direction  for  future  research. 
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SECTION  1 


INTRODUCTION 

Learning  new  and  involved  information  can  be  taxing  enough  without  the  added  pressure  of 
understanding  the  medium  of  instruction,  especially  in  a  computer  oriented  setting  (3).  A  computer-aided 
instruction  (CAI)  exercise  designed  in  accordance  with  human  factors  guidelines  can  help  learners  interact 
with  the  medium  with  reduced  stress  and  errors.  Therefore,  the  ease  with  which  learners  interact  with  the 
computer  can  contribute  to  the  efficiency  of  processing  students  through  computer  based  instruction.  Thus, 
user-computer  interface  (UCI)  is  an  important  concept  to  be  addressed  in  any  CAI  exercise  so  that  training 
efficiency  can  be  maximized. 

Recently  Hamel  and  Clark  (1)  developed  a  checklist  of  items  which  have  been  espoused  by  experts 
to  contribute  toward  the  ease  with  which  trainees  can  interact  with  CAI  systems.  Hamel  and  Clark's  (1) 
report  cites  numerous  studies  which  recognize  that  the  user-computer  interface  can  contribute  towards  the 
acceptance  of  CAI  by  the  user  in  order  to  maximize  the  utility  of  CAI  (See  Section  VI,  Bibliography). 

CAI  designed  with  the  best  instructional  technology  can  suffer  from  lack  of  acceptance  by  the  user 
community  due  to  a  poorly  designed  UCI. 

Poorly  designed  UCI  may  also  promote  processing  difficulties  for  the  trainee.  In  so  doing, 
interference  in  assimilating  information  can  have  negative  learning  affects  as  well  as  demotivating  the  user 
to  interact  with  a  frustrating  system.  Therefore,  UCI  may  directly  affect  training  gain  along  with  its  impact 
upon  training  efficiency  and  user  acceptance.  Since  the  trainee  is,  in  essence,  an  information  processor,  and 
since  information  processing  is  a  necessary  prerequisite  for  learning,  UCI  can  directly  affect  both  training 
gain  (learning)  and  training  efficiency.  However,  because  learning  is  dependent  upon  processing 
information  and  computer  interfaces  provide  the  source  of  information,  it  is  difficult  to  separate  the  impact 
of  this  information  on  learning  independent  from  its  effects  upon  training  efficiency. 

Tne  checklist  developed  by  Hamel  and  Clark  (1)  contains  human  factors  design  guidelines  for  CAI 
organized  into  five  categories  which  are  based  on  principles  of  UCI  suggested  by  Williges  and  Williges  (2). 
The  checklist  categories  -  brevity,  consistency,  flexibility,  compatibility,  and  responsiveness  -  contain 
guidelines  which  support  the  principle.  The  guidelines  were  produced  from  a  review  of  the  behavioral 
research  literature,  existing  UCI  guidelines,  and  verbal  reports  of  experienced  CAI  developers.  The 
categorized  checklist  items  were  distributed  to  one  expert  each  in  the  areas  of  computer  software,  education, 
and  human  factors  for  their  review  and  comments.  They  provided  constructive  comments  on  the  clarity  and 
importance  of  items  and  their  appropriateness  to  a  given  category.  The  authors  used  these  comments  to 
make  appropriate  modifications  to  the  checklist.  A  checklist  scoring  method  was  developed  which  allowed 
quantitative  measurement  of  those  qualities  of  the  UCI  represented  by  the  five  checklist  categories.  The 
principles  used  to  define  the  categories  are  described  below. 

Brevity  is  a  concept  that  deals  with  shortness  and  conciseness.  In  particular,  brevity  is  concerned 
with  minimizing  the  amount  of  information  to  be  attended  to. 

Consistency  enables  the  learner  to  predict  what  is  expected  of  him/her  and  is  of  particular  interest 
in  any  training  program.  The  format  and  the  location  on  the  screen  where  specific  kinds  of  information  are 
placed  are  kept  constant  In  this  way,  the  learner  should  be  able  to  relate  more  easily  to  the  task  at  hand 
instead  of  being  more  concerned  with  sudden  changes  in  the  format. 

Flexibility  helps  to  meet  the  different  and  changing  needs  of  the  learner.  Each  individual 
interacting  with  the  training  program  may  have  different  preferences  for  mode  of  interaction  as  well  as 
different  learning  abilities.  Thus,  flexibility  can  be  built  into  a  system  to  ensure  that  every  learner  has  an 
equal  opportunity  to  master  the  material  at  an  appropriate  pace  and  to  be  presented  the  material  in  an 
appropriate  sequence. 

Compatibility  refers  to  the  agreement  between  typical  expectations  and  the  manner  in  which 
information  is  presented.  The  learner  should  be  able  to  interact  with  the  computer  in  a  manner  which  fits 
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preestablished  conceptions.  For  example,  red  and  green  commonly  suggest  negative  and  positive  meanings 
(Le.,  incorrect/correct  responses).  Additionally,  text  is  typically  presented  in  a  left-right  sequence. 

Responsiveness  deals  with  providing  the  learner  with  informative  feedback  at  the  appropriate  time. 
It  also  gives  information  to  the  student  about  the  operations  of  the  system.  Responsiveness  can  be 
incorporated  into  a  training  program  so  that  every  individual  knows  where  he/she  stands  and  can  choose  a 
suitable  path  to  fit  one's  own  learning  needs. 

The  following  research  was  undertaken  to  meet  several  objectives:  (1)  to  evaluate  the  contribution 
of  these  five  principles  to  ease  of  interaction  with  a  CAI  program,  (2)  to  determine  the  relative 
effectiveness  of  characteristics  within  each  category  in  order  to  supply  weights  for  each  characteristic,  (3)  to 
determine  the  impact  of  characteristics  on  training  gain,  and  (4)  to  develop  a  handbook  for  CAI  designers 
describing  and  providing  examples  of  each  characteristic  along  with  a  scoring  technique  to  evaluate  CAI 
programs. 
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METHOD 


SUBJECTS 

Forty -five  subjects  having  some  college  background  participated  in  this  research  on  a  voluntary 
basis.  These  subjects  were  students  and/or  personnel  from  the  University  of  Central  Florida.  There  were  17 
males  and  28  females  between  the  ages  of  18  and  SO  years.  Two  subjects  were  excluded  because  they  failed 
to  complete  the  ratings.  Another  three  were  excluded  because  they  placed  maximum  subjective  ratings  in 
all  categories  for  all  characteristics  of  their  evaluations.  (It  was  felt  that  this  data  reflected  a  lack  of 
diligence  in  providing  evaluations  on  the  part  of  these  test  subjects.)  Two  additional  subjects  were 
eliminated  from  the  (raining  gain  part  of  the  experiment  as  they  obtained  perfect  pretest  scores  and  would 
therefore  not  show  any  training  gain. 


MATERIALS 

An  IBM  PC/XT  with  single  floppy,  hard  disk  drive,  640  KB  memory,  and  color  monitor  served  as 
the  training  workstation.  The  Trainer  Turned  Author  authoring  system  (distributed  by  Raster  Sciences, 
Inc.)  was  used  to  create  the  pretest  and  posttest  as  well  as  the  CAI  program. 

The  CAI  program  consisted  of  a  series  of  graphic  and  text  frames  developed  to  instruct  students  in 
logic  diagramming.  The  instruction  consisted  of  translating  verbal  statements  of  formal  logic  syntax  into 
graphic  diagrams  and  translating  graphic  diagrams  into  verbal  statements  of  formal  logic.  At  the  simplest 
level,  the  student  is  taught  how  to  translate  one  verbal  statement  of  formal  logic  into  a  graphic  diagram. 

An  example  of  a  typical  page  of  instruction  from  the  training  program  as  viewed  by  subjects  is  shown 
below  in  Figure  1. 


(Press  FI  To  Stop)  (Press  F2  To  Backup) 

2 

1:1  ’AH'  statements 

The  diagram  below  displays  the  statement 
'all  A’s  are  C's'. 

EXAMPLE  1  IS  CORRECT 
All  A's  are  C's  = 

The  entire  circle  A  is  within  circle  C. 

Thus,  'all  A's  are  C's'  is  shown. 

PRESS  ANY  KEY  TO  CONTINUE 


Figure  I .  A  Simple  Instructional  Frame  from  the  CAI  Program 


At  the  most  difficult  level,  the  student  is  instructed  how  to  translate  seven  verbal  statements  of  formal  logic 
into  a  graphic  diagram.  Figure  2,  shown  below,  is  another  example  from  the  training  program  depicting  a 


a 
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(Press  FI  to  Stop)  (Press  F2  To  Backup) 

88 

3:1  Multiple  statements 
STATEMENT  1  IS  CORRECT 
All  D's  Are  F's  And  H's  Equals  The  Below 


Circle  D  is  entirely  within  A,  C.  H,  and  F. 

Circle  D  could  have  been  drawn  anywhere 
within  all  of  C  and  still  conformed  to  the  above 
statements.  Thus,  this  answer  can  be  concluded 
by  the  "All  C’s  are  H’s"  statement. 

PRESS  ANY  KEY  TO  CONTINUE 


Figure  2.  A  Difficult  Instructional  Frame  from  the  CAI  Program 


The  instructional  sequence  progressed  from  the  simplest  to  the  most  difficult  as  the  examples  of 
Figure  1  and  Figure  2  demonstrate.  It  was  assumed  that  no  special  skills  were  required  of  the  college  level 
students  to  complete  the  course  of  instruction. 

A  questionnaire  incorporating  a  seven-  point  Likert  scale  for  each  characteristic  was  developed  in 
order  for  subjects  to  subjectively  weigh  the  contribution  of  each  characteristic  to  training  gain  and  to  ease  of 
use  of  the  CAI  program.  This  questionnaire  was  developed  to  also  serve  as  a  handbook  for  courseware 
designers  to  use  in  evaluating  their  CAI  programs  in  terms  of  user-computer  interface  design. 


PROCEDURE 

All  subjects  read  and  then  signed  a  consent  form  before  participating  in  the  research.  The  consent 
form  emphasized  that  a  subject  could  discontinue  participation  at  any  time  without  penalty  and  that  all 
personal  information  would  be  kept  strictly  confidential.  A  statement  of  standard  instructions  concerning 
the  objective  of  the  experiment  was  given  to  each  subject  to  read  prior  to  participation. 

First,  each  subject  took  a  six-question  pretest  to  assess  h^er  prior  knowledge  of  logic 
diagramming,  the  topic  presented  in  the  CAI  program.  Subjects  then  underwent  the  training  program  which 
embedded  the  various  UCI  characteristics  presented  in  Appendix  A.  Subjects  were  instructed  to  go  through 
each  of  the  four  levels  of  the  training  program  sequentially  to  ensure  that  they  would  view  all  of  the  UCI 
characteristics.  Following  completion,  they  took  a  posttest  identical  to  the  pretest  to  assess  their  gain  in 
knowledge. 

Following  the  posttest,  each  test  subject  then  paged  through  the  questionnaire  of  UCI  features  (see 
Appendix  A).  Each  subject  was  presented  with  the  categories  of  UCI  characteristics  in  one  of  five 
counterbalanced  orders  so  as  to  minimize  any  order  of  presentation  effects  of  questionnaire  items.  For  each 
item  of  the  questionnaire,  subjects  judged  the  contribution  of  each  feature  by  circling  a  number  ranging 
from  one  to  seven  on  a  Likert  scale.  Each  item  was  judged  for  contribution  towards  ease  of  use  of  the 
system.  Following  judgements  on  all  characteristics  for  ease  of  use,  subjects  were  instructed  to  record  their 
judgements  on  each  characteristic's  contribution  to  training  gain. 
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ANALYSIS 


The  simultaneous  model  for  multiple  regression  correlation  (MRC)  analysis  was  selected  for 
purposes  of  data  analysis  (4).  This  model  is  most  appropriately  used  as  compared  to  a  hierarchical  or 
stepwise  model  when  there  is  no  a  priori  rationale  for  ordering  variables  in  terms  of  their  importance  (5). 

The  first  set  of  analyses  is  analogous  to  an  item  analysis  whereby  item  to  total  correlations  are  obtained  (6). 
The  ratings  for  each  characteristic  in  each  category  were  correlated  to  the  sum  totals  of  all  ratings  for  all 
subjects.  The  sum  totals  of  all  ratings  for  each  subject  were  converted  to  z  scores.  This  set  of  analyses  was 
conducted  to  determine  the  contribution  of  each  characteristic  within  each  category  to  the  total  of  all  ratings. 

A  second  set  of  analyses  was  conducted  to  compare  ease  of  use  and  training  gain  category  ratings  to 
actual  training  gain.  First,  the  ratings  were  averaged  for  each  category.  Then  these  averages  were  used  in  a 
multiple  regression  analysis  which  computed  the  relative  contribution  of  each  category  variance  to  actual 
training  gain.  Two  multiple  regression  analyses  were  conducted,  one  on  the  ease  of  use  ratings  and  the 
second  on  the  training  gain  ratings. 

A  power  analysis  was  also  conducted  to  determine  the  number  of  test  subjects  needed  to  obtain 
reliable  results  for  large  effect  sizes.  In  an  analysis  of  partial  regression  correlation  coefficients,  a  power  of 
.80  can  be  obtained  for  large  effect  sizes  employing  40-50  test  st.ojects.  This  determination  of  power  was 
calculated  employing  the  conventions  described  by  Cohen  (7).  Additionally,  a  post  hoc  power  analysis  was 
conducted  to  aid  in  evaluating  the  potential  for  finding  other  characteristics  which  might  yield  significant 
results  given  that  more  test  subjects  are  run.  This  post  hoc  analysis  allows  the  experimenter  to  determine  if 
characteristics  having  a  medium  effect  size  might  prove  to  be  significant  in  the  event  that  additional  test 
subjects  are  run  or  further  research  is  conducted. 

In  the  present  context,  a  large  effect  size  is  associated  with  a  partial  variance  of  approximately  .20. 
Therefore,  effect  sizes  approximating  .20  would  have  a  power  of  .80  if  approximately  40-50  test  subjects 
were  run.  A  medium  effect  size,  in  this  context,  is  associated  with  a  partial  variance  of  approximately  .09. 
To  obtain  a  power  of  .80,  given  this  effect  size,  approximately  90-100  test  subjects  would  have  to  be  run. 

A  small  effect  size,  in  this  context,  is  associated  with  a  partial  variance  of  .02.  To  obtain  a  power  of  .80, 
given  this  effect  size,  approximately  400  test  subjects  would  have  to  be  run.  To  have  reliable  large  effects, 
in  this  study,  45  subjects  were  tested.  To  reliably  detect  medium  and  small  eff  xt  sizes  among  the 
characteristics  identified,  an  N  of  approximately  90-100  and  400  would  be  required,  respectively.  Running 
these  large  numbers  of  subjects  was  beyond  the  scope  of  this  initial  effort.  However,  by  examining 
characteristics  which  have  medium  effect  sizes,  experimenters  can  get  some  insight  as  to  the  likelihood  of 
certain  characteristics  proving  to  be  significant  upon  further  research  or  extensions  of  the  present  research. 
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SECTION  m 


RESULTS 

TRAINING  GAIN 

On  the  average,  there  was  a  52%  improvement,  as  measured  by  pretest  and  posttest  scores  by 
subjects  who  completed  the  CAI  logic  diagramming  program.  The  mean  training  gain  for  pretest  scores 
was  52.58  and  for  posttest  scores  was  79.82,  with  a  standard  deviation  of  25.99  and  20.85,  respectively. 

RATINGS  OF  EASE  OF  USE 

The  mean  scores  of  ratings  of  the  importance  of  checklist  items  to  ease  of  use  ranged  from  4.35  to 
6.35.  Table  B-l  provides  the  means  and  standard  deviations  of  all  ease  of  use  ratings.  The  sums  of  all 
ratings  for  each  subject  were  converted  to  z  scores.  These  z  scores  were  used  in  the  multiple  regression 
analyses  performed  on  the  ease  of  use  data. 

Full  model  multiple  regression  analyses  (4)  were  conducted  to  determine  the  contribution  of 
characteristics  to  the  total  ease  of  use  variance.  Likert  Scale  ratings  for  each  characteristic  within  each 
category  were  correlated  with  the  z  scores  obtained  for  each  subject.  Five  multiple  regression  analyses  were 
conducted,  one  for  each  of  the  five  categories.  Tables  1-5  show  the  results  of  these  analyses.  All  of  the 
categories  had  a  multiple  R  significant  alp  <  .05,  indicating  that  the  variances  of  all  categories  contributed 
significantly  to  the  overall  variation. 

Effect  sizes  are  represented  by  the  partial  r2's  in  the  last  column.  According  to  a  power  analysis, 
several  items  demonstrated  large  effects.  That  is,  they  contributed  more  than  20%  (approximately)  of  their 
category  variance.  These  items  are  marked  by  an  asterik  in  the  tables.  Several  items  demonstrated  medium 
effects.  That  is,  they  contributed  more  than  9%  (approximately)  of  their  category  variance.  These  items  are 
marked  by  a  crossbar  in  the  tables.  The  coefficients  in  the  first  column,  which  are  directly  correlated  with 
the  partial  r^’s  are  the  Beta  weights  used  to  derive  the  weighted  scoring  method  for  the  CAI  Evaluation 
Handbook  (8). 
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Adjusted  R  Squared  =0.84 
R  Squared  =  0.88 
Multiple  R  =  0.94 


*  =  demonstrate  large  effect  sizes  (contribute  more  than  approx.  20%  of  their  category  variance) 
t  =  demoastrate  medium  effect  sizes  (contribute  more  than  approx.  9%  of  their  category  variance) 
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Source 

Sum  of  Squares 
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F  Ratio 

Prob. 
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1.25E-10 
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TABLE  2.  CONTRIBUTION  OF  INDIVIDUAL  CHARACTERISTICS  OF  CONSISTENCY  TO  EASE 
OF  INTERACTION 


Characteristic _ 

Functionally  alike  screens  are  formatted 
in  the  same  way 
When  erased,  functional  areas  are 
rewritten  in  the  same  order 
Consistent  use  of  labels  and  graphics 
Critical  information  comes  at  beginning 
of  message  or  centered  on  screen 
Constant  delay  of  feedback 
Similarity  in  the  way  questions  are 
asked  and  responses  are  made 
Overall  structure  is  clear  through 
use  of  menus  and  maps 
A  symbol  always  has  the  same  meaning 
Input  prompts  are  always  in  the 
same  area  of  display 

Page  numbers  shown  in  upper  right-hand 
comer  for  multiscreen  transactions 
CONSTANT 

Adjusted  R  Squared  =0.79 
R  Squared  =  0.84 
Multiple  R  =  0.92 


_ Regression  Analysis 

Coefficient  Std.  Error  T  (DF=29) 


Prob.  Partial  t*2 


0.02 

0.15 

0.14 

.89 

0.01 

0.31 

0.18 

1.74 

.09 

0.09  t 

0.01 

0.11 

0.12 

.90 

0.01 

0.12 

0.11 

1.03 

.31 

0.04 

-0.08 

0.09 

-0.85 

.40 

0.02 

0.18 

0.13 

1.35 

.19 

0.06 

0.06 

0.06 

0.98 

.34 

0.03 

0.20 

0.10 

2.05 

.05 

0.13  t 

0.10 

0.12 

0.78 

.44 

0.02 

0.04 

0.05 

0.93 

.36 

0.03 

_ Analysis  of  Variance _ 

Source  Sum  of  Squares  DF  Mean  Square 

Regression  32.86  10  3.29 

Residual  6.14  29  0.21 

Total  39.00  39 


t  =  demonstrate  medium  effect  sizes  (contribute  more  than  approx.  9%  of  their  category  variance) 


F  Ratio  Prob. 
15.53  4.13E-09 
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TABLE  3.  CONTRIBUTION  OF  INDIVIDUAL  CHARACTERISTICS  OF  FLEXIBILITY  TO  EASE 
OF  INTERACTION 


Regression  Analysis 

Characteristic 

Coefficient 

S  td.  Error 

T(DF=29) 

Prob. 

Partial  1*2 

Students  can  page  back  to  review 

0.16 

0.12 

1.31 

.202 

0.06 

Students  can  exit  lessons,  return  to 

0.07 

0.09 

0.72 

.478 

0.02 

menus,  and  exit  the  program 

Student  has  control  over  rate  of 

0.19 

0.13 

1.52 

.138 

0.07 

presentation  of  frames 

Student  can  request  more  lengthy 

-0.14 

0.16 

-0.89 

.381 

0.03 

messages  for  further  clarification 

Activities  for  diagnosis  of  skills 

0.20 

0.12 

1.68 

.104 

0.09  t 

already  mastered 

Remedial  exercises  for  skill  deficiencies 

0.06 

0.14 

0.42 

.678 

0.01 

Modularized  program  allows  student 

-0.00 

0.12 

-0.03 

.979 

0.01 

to  begin  at  appropriate  place 

Student  can  choose  difficulty  level  of 

0.09 

0.11 

0.84 

.408 

0.02 

problems  or  exercises 

Student  can  correct  input  errors 

-0.12 

0.08 

-1.53 

.137 

0.07 

Student  can  choose  an  important  option 

0.36 

0.09 

4.17 

.001 

0.37  * 

and  implement  it  at  any  time 

CONSTANT 

-5.04 

Adjusted  R  Squared  =0.75 

Analysis  of  Variance 

Source 

Sum  of  Squares  DF  Mean  Square 

F  Ratio  Prob. 

R  Squared  =  0.81 

Regression 

31.61 

10 

3.16 

12.41  5.27E-08 

Multiple  R  =  0.90 

Residual 

7.39 

29 

0.25 

Total 

39.00 

39 

*  =  demonstrate  large  effect  sizes  (contribute  more  than  approx.  20%  of  their  category  variance) 
t  =  demonstrate  medium  effect  sizes  (contribute  more  than  approx.  9%  of  their  category  variance) 


m 


s 


m 


m 


5^7 


NTSC  TR87-030 


TABLE  4.  CONTRIBUTION  OF  INDIVIDUAL  CHARACTERISTICS  OF  COMPATIBILITY  TO  EASE 
OF  INTERACTION 


Regression  Analysis 


info; 

Prob. 

Partial  i*2 

Response  mode  is  appropriate  to 

0.06 

0.07 

0.87 

.395 

0.03 

audience 

Students  are  required  to  use  codes  for 

0.08 

0.07 

1.20 

.242 

0.05 

responding  only  when  necessary 

Visual  information  and  tasks  are 

0.06 

0.08 

0.77 

.446 

0.02 

presented  graphically 

Where  frames  are  labeled,  title,  not 

-0.03 

0.03 

-0.81 

.423 

0.03 

number  is  used  for  identification 

Input,  output  is  consistent  with  user 

0.06 

0.04 

1.36 

.186 

0.07 

population  stereotypes 

Menu  options  are  listed  by  number 

0.29 

0.06 

4.75 

.001 

0.47  * 

where  order  of  lessons  is  important 

A  sample  item  is  answered  before  quiz 

0.23 

0.06 

4.09 

.001 

0.40  * 

to  clarify  drill  or  test  instructions 

Response  is  demanded  while  instructions 

0.08 

0.07 

1.27 

.216 

0.06 

are  on  screen 

Routing  menus  are  limited  to  three  levels 

0.06 

0.10 

0.59 

.563 

0.01 

Text  is  displayed  row  by  row 

0.04 

0.12 

0.33 

.748 

0.01 

Opposite  colors  are  used  to  make  items 

0.03 

0.05 

0.75 

.458 

0.02 

distinct 


(table  continues) 


."JVDvV 


Characteristic 


Graphics  arc  used  for  further  clarification 


of  text 


Menu  selections  arc  left  justified 


and  in  columns 


Directions  come  before  menu  selections 


CONSTANT 


Coefficient  Std.  Error  T  (DF=25)  Prob.  Partial  i*2 


0.15  t 


Analysis  of  Variance 


Source  Sum  of  Squares  DF  Mean  Square  F  Ratio  Prob. 


Adjusted  R  Squared  =  0.92 
R  Squared  =  0.95 
Multiple  R  =  0.97 


*  =  demonstrate  large  effect  sizes  (contribute  more  than  approx.  20%  of  their  category  variance) 
t  =  demonstrate  medium  effect  sizes  (contribute  more  than  approx.  9%  of  their  category  variance) 
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TABLE  5.  CONTRIBUTION  OF  INDIVIDUAL  CHARACTERISTICS  OF  RESPONSIVENESS  TO  EASE 
OF  INTERACTION 


Regression  Analysis 

Characteristic 

Coefficient 

Std.  Error 

Prob. 

Partial  r*2 

Periodic  feedback  indicates  normal 

0.14 

0.08 

1.74 

.09 

0.09  t 

operation  when  waiting 

Computer  tracks  response  patterns 

-0.01 

0.10 

-0.13 

.90 

0.01 

and  gives  option  to  pursue  remediation 

Feedback  and  directions  are 

0.09 

0.05 

1.80 

.08 

0.10  t 

distinguishable  from  other  text 

At  higher  levels,  more  lengthy  feedback 

-0.03 

0.11 

-027 

.79 

0.01 

is  delayed  until  end  of  session 

Pause  after  feedback  allows 

0.25 

0.10 

2.54 

.02 

0.18  * 

consolidation  of  material 

Access  to  helps,  references,  or  resources 

0.15 

0.09 

1.70 

.10 

0.09  t 

are  easily  available 

Feedback  is  response  specific  at 

0.14 

0.10 

1.35 

.19 

0.06 

beginning  of  training 

Takes  no  more  than  5  seconds  for  text 

0.05 

0.09 

0.62 

.54 

0.01 

and  graphics  to  fill  screen 

More  than  one  chance  to  give  answer 

0.21 

0.07 

2.84 

.01 

0.21  * 

CONSTANT 

-5.57 

Adjusted  R  Squared  =0.81 

Analysis  of  Variance 

Source 

Sum  of  Squares  DF  Mean  Square 

F  Ratio  Prob. 

R  Squared  =  0.85 

Regression 

33.27 

9 

3.70 

19.36  3.48E-10 

Multiple  R  =  0.92 

Residual 

5.73 

30 

0.19 

Total 

39.00 

39 

— 

*  =  demonstrate  laige  effect  sizes  (contribute  more  than  approx.  20%  of  their  category  variance) 
t  =  demonstrate  medium  effect  sizes  (contribute  more  than  approx.  9%  of  their  category  variance) 


NTSC  TR87-030 


LARGE  EFFECT  SIZES 


In  accordance  with  the  power  constraints  placed  upon  the  design  and  analysis  of  the  experiment, 
those  significant  results  for  large  effect  sizes  (i.e.,  >  .20)  are  shown  below. 


Ease  of  Use  Ratings  on  Individual  Characteristics  Related  to  Overall  Ease  of  Use  Score 


The  significant  large  effect  size  characteristics  for  ease  of  use  ratings  on  individual  characteristics 
related  to  overall  ease  of  use  scores  are  shown  below. 


Brevity.  "Text  broken  into  meaningful  chunks"  was  found  to  account  for  19%  of  Brevity's 
contribution  to  ease  of  interaction  ip  <  .015).  Also  "No  more  than  3-4  screens  without  interactivity" 
accounted  for  28%  of  Brevity's  contribution  to  ease  of  interaction  (p  <  .002). 

Flexibility.  "Student  can  choose  an  important  option  and  implement  it  at  any  time"  accounted  for 
37%  of  Flexibility's  contribution  to  ease  of  interaction  ip  <  .001). 

Compatibility.  "Menu  options  are  listed  by  number  where  order  of  lessons  is  important"  and  "A 
sample  item  is  answered  before  quiz  to  clarify  drill  or  test  instructions"  accounted  for  47%  (p  <  .001)  and 
40%  {p  <  .001),  respectively,  of  Compatibility's  contribution. 

Responsiveness.  "Pause  after  feedback  allows  consolidation  of  material"  and  "More  than  one 
chance  to  give  answer”  accounted  for  18%  ip  <  .02)  and  21%  ip  <  .01),  respectively,  of  Responsiveness's 
contribution  to  ease  of  interaction. 

MEDIUM  EFFECT  SIZES 

Some  characteristics  that  are  categorized  under  a  medium  effect  size  do  meet  significance  criterion 
and  some  do  not.  Those  which  do  meet  an  alpha  criterion,  however,  do  not  meet  beta  criterion  or  power 
criterion.  Therefore,  their  reliability  is  questionable.  Due  to  the  nature  of  this  screening  experiment,  it  is 
felt  that  medium  effect  sizes  are  worthy  of  further  research  consideration  employing  a  larger  number  of 
subjects. 


Ease  of  Use  Ratings  on  Individual  Characteristics  Related  to  Overall  Ease  of  Use  Score 


The  medium  effect  size  characteristics  of  ease  of  use  ratings  on  individual  characteristics  related  to 
overall  ease  of  use  scores  are  shown  below. 

Brevity.  "Data  entries  are  no  more  than  8-10  characters"  accounted  for  12%  ip  <  .052)  of  brevity's 
contribution  to  ease  of  use. 

Consistency.  "When  erased,  functional  areas  are  rewritten  in  the  same  order"  and  "A  symbol 
always  has  the  same  meaning”  accounted  for  9%  ip  <  .09)  and  13%  ip  <  .05)  of  consistency’s  contribution 
to  ease  of  interaction. 

Flexibility.  "Activities  for  diagnosis  of  skills  already  mastered"  accounted  for  9%  ip  <  .104)  of 
flexibility's  contribution  to  ease  of  interaction. 

Compatibility.  "Menu  selections  are  left  justified  and  in  columns"  accounted  for  15%  ip  <  .046) 
of  compatibility's  contribution  to  ease  of  interaction. 

Responsiveness.  "Periodic  feedback  indicates  normal  operation  when  waiting", "  Feedback  and 
directions  are  distinguishable  from  other  text",  and  "Access  to  helps,  references,  or  resources  are  easily 
available"  accounted  for  9%  ip  <  .09),  10%  ip  <  .08),  and  9%  (p  <  .100)  of  responsiveness's  contribution 
to  ease  of  use,  respectively. 
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COMPARISONS  OF  EASE  OF  USE  AND  TRAINING  GAIN  RATINGS 

The  mean  scores  of  ratings  of  the  importance  of  checklist  items  to  training  gain  ranged  from  3.24 
to  6.63.  Table  B-2  provides  the  means  and  standard  deviations  of  training  gain  ratings. 

These  ratings  were  used  in  a  multiple  regression  analysis  which  averaged  ratings  in  each  category 
and  then  looked  at  the  relative  contribution  of  the  category  to  actual  training  gain.  The  same  type  of 
regression  analysis  was  done  with  the  ease  of  use  ratings,  so  that  the  two  could  be  compared  on  their 
relationship  to  training  gain.  The  results  are  shown  in  Tables  6  and  7. 

Table  6  indicates  that  the  responsiveness  category  contributed  a  significant  amount  of  the  variance 
(p  <  .02)  accounting  for  17%  of  the  overall  variation.  The  correlation  was  negative,  Le.,  the  ratings  of  the 
importance  of  responsiveness  to  training  gain  were  inversely  related  to  actual  training  gain. 

Table  7  indicates  that  two  categories,  responsiveness  and  consistency,  contributed  a  significant 
portion  of  the  variance  (p  <  .02)  accounting  for  24%  and  17%  of  the  overall  variation,  respectively.  The 
correlations  were  negative,  Le.,  the  ratings  of  the  importance  of  responsiveness  and  consistency  to  ease  of 
use  were  inversely  related  to  actual  training  gain. 


NTSC  TR87-030 


TABLE  &  CONTRIBUTION  OF  TRAINING  GAIN  RATINOS  OF  MAIN  CATEGORIES  TO  TRAIN3NO  GAIN 


Analysis  of  Variance 


Source  Sum  of  Squares  DF  Mean  Square  F  Ratio 


Regression 


2.49  3.24 


Multiple  R  =  0.58 


Residual 


24.57 


32 


0.77 
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SECTION  IV 


DISCUSSION 

The  major  issue  of  this  research  was  the  application  of  five  human  factors  principles  to  the  design 
of  the  user  interface  for  CAL  A  CAI  program  on  logic  diagramming  was  developed  using  characteristics  of 
user  interface  design  taken  from  Hamel  and  Clark  (1).  Subjects  who  completed  the  lessons  showed  an 
average  of  52%  improvement  as  measured  by  pretest  and  posttest  scores. 

Subjects  were  asked  to  rate  the  importance  of  interface  characteristics  used  to  design  the  CAI 
program  after  they  completed  the  lessons.  The  results  of  the  analyses  of  subjects'  "ease  of  use"  ratings  of 
the  CAI  system  indicated  that  each  of  the  categories  incorporated  into  the  ratings  checklist  significantly 
contributed  to  the  ease  of  use  variance.  Comparisons  of  mean  ratings  for  each  principle  revealed 
insignificant  differences  between  categories.  The  results  suggest  that  the  categories  or  principles  of  brevity, 
consistency,  flexibility,  compatibility,  and  responsiveness  all  significantly  contributed  to  ease  of  use  of  a 
CAI  system. 

In  conjunction  with  this  research,  a  handbook  was  developed  to  explain  the  user  interface 
characteristics  and  provide  examples.  The  evaluation  checklist,  first  developed  by  Hamel  and  Clark  (1),  was 
included  in  the  handbook  in  a  revised  form.  The  checklist  is  intended  to  be  used  as  a  way  to  determine  if 
known  user  interface  characteristics  have  been  incorporated  appropriately  into  a  CAI  system.  Based  on  the 
number  of  characteristics  that  have  been  implemented,  scores  can  be  obtained  for  each  of  the  five  categories 
listed  above.  With  the  assumption  that  some  items  in  the  checklist  contribute  more  than  others  to  ease  of 
use,  a  weighted  scoring  method  was  desired  to  produce  more  accurate  assessment  measures.  Subjects  in  the 
experiment  were  given  the  revised  checklist  as  part  of  a  handbook  to  use  to  rate  the  user  interface  of  the  CAI 
logic  diagramming  program.  Weightings  indicating  the  relative  contributions  of  individual  items  were 
derived  from  the  multiple  regression  analyses  performed  on  the  "ease  of  use”  ratings.  The  Beta  coefficients 
shown  in  Tables  1-5  were  used  to  assign  weights  to  the  items  in  the  checklist,  and  this  revised  scoring 
method  has  been  incorporated  into  the  CAI  Evaluation  Handbook  (8). 

Based  on  a  power  analysis,  the  majority  of  characteristics  found  to  have  made  a  significant 
contribution  to  their  respective  categories  had  large  effect  sizes;  a  few  had  medium  effect  sizes.  Eight 
additional  characteristics  had  medium  effect  sizes  but  were  not  significant.  A  majority  of  the  characteristics 
demonstrating  a  medium  effect  may  have  been  significant  if  another  forty  or  fifty  test  subjects  were  run 
yielding  a  power  value  of  .80.  It  is  suggested  that  checklist  items  with  both  medium  and  large  effect  sizes 
be  given  special  attention  when  conducting  UCI  research  and  evaluation.  Future  research  is  necessary  to 
ascertain  the  importance  of  the  characteristics  demonstrating  a  medium  effect  size  as  a  result  of  this 
experiment 

The  remaining  characteristics,  which  demonstrate  small  effect  sizes,  are  nonsignificant;  however, 
the  weights  derived  provide  a  best  estimate  of  their  relative  contribution  to  ease  of  use.  When  all  of  the 
characteristics  in  a  category  are  pooled  together,  the  category  contributes  significantly  to  the  ease  of  use 
variance.  New  interface  technologies  are  continuously  under  development.  The  contributions  of  new 
characteristics  derived  from  these  technologies  may  have  greater  effect  sizes.  Future  research  efforts  may 
produce  a  greater  return  by  focusing  upon  more  recent  technological  developments,  rather  than  studying  the 
subtleties  represented  by  the  checklist  items  with  small  effect  sizes. 

Further  analyses  were  conducted  to  determine  if  perceived  ease  of  use  is  related  to  actual  training 
gain  in  CAI.  This  question  assu"'?:  ’hat  a  system  which  is  easier  to  use  shall  reduce  frustration  and 
interference  in  learning  acthiues  leading  to  improved  training  gain.  Several  characteristics  listed  in  the 
checklist  could  be  inferred  to  represent  both  a  learning  principle  and  an  interface  property,  increasing  the 
likelihood  that  ease  of  use  would  be  significantly  correlated  with  actual  training  gain.  It  was  found  that 
both  ratings  of  responsiveness  and  consistency  were  negatively  correlated  with  actual  training  gain.  That  is, 
the  lesser  the  actual  training  gain  accomplished  during  the  lessons,  the  higher  were  the  ratings  of  the 
perceived  importance  of  responsiveness  and  consistency  to  ease  of  use.  It  may  be  that  subjects  who  were 
having  trouble  learning  the  CAI  material  were  most  in  need  of  a  good  user  interface,  and  so  rated  the 
characteristics  as  more  important.  Along  these  lines,.other  investigators  (9)  have  found  an  interaction 
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between  student  aptitude  and  training  material  format.  In  a  controlled  study,  it  was  demonstrated  that  those 
with  lower  aptitudes  benefited  from  the  experimental  formats  more  than  higher  aptitude  trainees.  Those 
with  higher  aptitudes  learned  better  with  the  experimental  formats,  but  not  to  the  degree  that  the  lower 
aptitude  trainees  did.  Future  research  on  UCI  characteristics  requires  controlled  experiments  to  test  the 
actual  contribution  of  UCI  characteristics  to  learning.  More  objective  measures  of  ease  of  use  will  be 
needed.  Such  measures  could  include  human  information  speed,  indicated  by  reaction  time,  as  employed  by 
Card,  Moran,  and  Newell  (10)  in  their  Goals  Operations  Methods  and  Selection  (GOMS)  model.  Another 
measure  could  be  cognitive  complexity  measured  by  the  number  of  productions  required  to  efficiently 
interact  with  the  device,  as  proposed  by  Kieras  and  Poison  (1 1). 

It  is  not  surprising  that  the  ratings  of  the  importance  of  responsiveness  to  ease  of  use  were  found 
to  be  correlated  with  actual  training  gain.  Several  characteristics  in  that  category  pertain  to  feedback,  which 
is  obviously  related  to  learning.  In  another  set  of  analyses  which  correlated  ratings  of  perceived  training 
gain  to  actual  training  gain,  similar  results  were  found.  Subjects'  ratings  of  the  importance  of 
responsiveness  to  perceived  training  gain  were  found  to  be  negatively  correlated  with  actual  training  gain. 

The  similarity  of  subject  ratings  on  ease  of  use  and  training  gain  suggest  that  subjects  did  not 
distinguish  the  characteristics  along  these  dimensions.  In  the  special  case  of  the  UCI  for  CAI  systems,  it 
may  be  impossible  to  separate  these  two  properties.  Future  research  must  take  into  account  the  overlap  of 
the  human  factors  principles  described  in  this  research  and  well-documented  learning  principles.  The  human 
factors  principles,  based  on  human  information  processing  theory,  complement  well-established  theories  of 
human  learning. 

In  conclusion,  the  research  provides  validation  of  the  human  factors  principles  of  user  interface 
design  proposed  by  Williges  and  Williges  (2).  The  research  also  provides  a  weighted  scoring  method  for  a 
CAI  evaluation  checklist  Statistical  analyses  aimed  at  assessing  the  importance  of  individual  checklist 
items  revealed  directions  for  future  research.  An  issue  of  concern  for  future  UCI  research  is  the  obvious 
overlap  of  learning  principles  and  interface  design  principles  based  on  human  information  processing  theory. 
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APPENDIX  A 


CAI  HANDBOOK/CHECKLIST  ITEMS 

The  questionnaire  was  designed  to  obtain  subjective  evaluations  of  the  training  program.  It  was 
specifically  designed  to  acquire  reactions  to  the  contributions  of  certain  program  characteristics  towards  ease 
of  interaction.  The  questionnaire  was  also  developed  to  serve  as  a  handbook  describing  and  providing 
examples  of  how  each  characteristic  could  be  implemented  in  a  CAI  program.  This  handbook  can  be  used 
by  designers  to  evaluate  the  UCI  of  CAI  programs  which  they  create.  The  handbook/questionnaire  was 
divided  into  the  five  major  categories  of  brevity,  consistency,  flexibility,  compatibility,  and  responsiveness. 
Each  category  has  certain  characteristics  contained  within  it  The  handbook/questionnaire  contains 
descriptions  of  the  specific  characteristics  as  well  as  of  the  five  major  categories.  Each  characteristic  is 
accompanied  by  an  example  of  screens  taken  directly  from  the  training  program.  A  pair  of  seven-point 
Likert  scales  was  provided  on  the  bottom  of  each  page  which  described  a  characteristic;  this  allowed  the 
subjects  to  subjectively  evaluate  the  characteristic's  contribution  towards  ease  of  interaction  and  training 
gain.  Figure  A-l  provides  an  example  page  Grom  the  handbook. 

The  following  is  a  list  of  those  characteristics  compiled  by  Hamel  and  Clark  (1)  plus  some 
additional  characteristics  grouped  by  their  respective  categories. 

Bsyitt 

-  Large  portions  of  text  are  broken  into  meaningful  "chunks."  This  minimizes  the 
amount  of  information  to  be  attended  to  at  one  time. 

-  No  more  than  seven  to  eight  lines  of  text  per  screen. 

-  Graphics  displays  take  up  15%  to  25%  of  the  screen  area. 

-  Main  menus  and  submenus  have  no  more  than  five  to  nine  choices. 

-  Use  of  color,  boxing,  highlighting,  and  text  style  rather  than  blinking  to  focus 
attention  on  important  items. 

-  No  more  than  three  or  four  text  screens  without  interactivity. 

-  The  time  required  for  a  typical  session  (or  lesson)  is  within  the  attention  span  of  the 
target  audience. 

-  Sentences  have  simple  syntax:  active  voice,  not  compounded. 

-  Data  entries  by  the  student  are  no  more  than  eight  to  ten  characters. 

-  The  field  width  for  each  line  is  40  characters  or  less. 


Consistency 

-  Functionally  alike  screens  are  formatted  in  the  same  way. 

-  When  functional  areas  are  erased,  they  are  consistently  rewritten  in  the  same  order. 
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-Consistent  uaeoflabelsandgrapfaicskccps  the  ramc  type  of  frames  identified  as 
fnr4i  Directions,  instructions,  example,  and  question  screens  each  have  cheir 
distinctive  format 

-  Critical  information  is  always  presented  at  the  beginning  of  a  message  and  centered 
an  the  screen. 

-  Students  receive  constant  delay  of  feedback  (no  more  than  two  seconds),  rather  than 

variable  delays. 

•  Similarity  in  the  way  questions  are  asked  and  similarity  in  die  way  responses  are 
made. 

-  The  structure  of  the  presentation  is  evident  to  the  user  through  the  use  of  menus  and 

concept  maps. 

-  A  symbol  has  the  same  meaning  all  the  time. 

•  Input  prompts  are  positioned  in  the  same  area  of  display  consistently. 

•  A  page  number  is  always  shown  in  the  upper  right-hand  comer  of  the  display  for 
multiscreen  transactions. 


Flexibility 

-  A  page-back  capability  allows  the  student  to  review  previous  material. 

-  Students  can  easily  exit  lessons,  return  to  menus,  and  exit  the  program. 

-  The  student  has  control  over  the  rate  of  presentation  of  frames. 

-  The  student  can  request  more  lengthy  messages  if  further  clarifications  arc  needed. 

-  The  program  contains  activities  for  diagnosis  of  skills  already  mastered. 

-  There  are  remedial  exercises  for  skill  deficiencies. 

-  Modularized  program  (with  menus)  allows  the  student  to  begin  at  a  point 
appropriate  to  past  achievement. 

-  The  student  can  choose  the  difficulty  level  of  problems  or  exercises. 

-  The  student  can  correct  an  input  error  (e.g„  with  BACKSPACE)  or  recover  from 

input  errors  without  disrupting  the  lesson  sequence. 

-  The  student  is  able  to  choose  an  option  that  is  used  often  or  is  of  critical 

importance  and  implement  it  at  anytime. 


Compatibility 

-  The  response  mode  is  appropriate  to  the  target  audience.  Research  has  found  that 
information  that  is  presented  auditorily  is  cognitively  compatible  with  verbal 
responses.  Likewise,  information  that  is  presented  spatially  is  cognitively 
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compatible  with  motor  responses.  Thus,  in  a  CA1  program,  motor  responses  are 
appropriate. 

-  Students  arc  required  to  use  codes  for  responding  only  when  necessary,  as  in 
multiple  choice  answering  (eg„  l»=yes,  2=no,  is  unnecessary  coding). 

-  Visual  infonnation  and  visual  fades  such  as  locating  or  repositioning  are  presented 
graphically.  The  trainee  is  asked  in  a  CAI  program  to  respond  to  information 
which  is  primarily  visual,  and  consequently  graphic  or  pictorial  information  should 
be  presented  throughout 

-  'Where  frames  are  labeled,  title,  not  number,  is  used  for  identification. 

•  Input  output  is  consistent  with  user  population  stereotypes  (e.g.  correct  response 
feedback  is  in  green). 

-  Where  order  of  lessons  is  important  menu  options  are  listed  by  number,  not  by 
letter. 

-  To  clarify  drill  or  test  instructions,  a  sample  item  is  answered  before  the  drill  or 
quiz  begins. 

-  A  response  is  demanded  while  instructions  on  how  to  respond  are  still  on  the 

screen. 

-  Routing  menus  are  limited  to  a  maximum  of  three  levels. 

-  Text  is  displayed  row  by  tow,  not  in  column  formations. 

-  To  make  items  distinct  and  separate  from  one  another,  opposite  colors  arc  used. 

-  Graphics  and  illustrations  are  used  for  further  clarification  of  text. 

-  Menu  selections  available  to  the  student  are  left  justified  and  in  column  formation. 

-  Directions  always  come  before  the  menu  selections. 


Responsiveness 

-  When  the  student  must  stand  by,  periodic  feedback  indicates  normal  operation. 

-  The  computer  tracks  response  patterns  and  gives  the  student  the  option  to  pursue 

further  remediation  if  desired. 

-  Feedback  and  directions  are  clearly  distinguishable  from  other  text  through  the  use 
of  color,  boxing,  reverse  video,  etc. 

-  At  higher  mastery  levels,  students  are  given  immediate  knowledge  of  right  and 

wrong  responses,  and  more  lengthy  feedback  is  delayed  until  the  end  of  the  session. 

-  There  is  a  pause  after  feedback,  before  the  lesson  continues,  to  allow  time  for 
consolidation  of  the  newly  acquired  material. 

-  Access  to  helps,  references,  or  resources  are  easily  available. 
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At  the  beginning  of  training,  feedback  is  response  specific,  (e^,  "tbc  —  part  of 
year  answer  Is  incorrect") 

It  takes  no  more  than  five  seconds  for  text  and  graphics  to  fill  the  screen. 

The  student  gets  more  than  one  chance  to  give  the  answer  (with  prompts). 
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Fig.  A-l.  Example  Page  From  Handbook 
Examples: 


;  FI  To  Stop) 


(Press  F2  To  Backup) 
14 


Some  A’s  are  C's  =  ? 

dQ  ’O,*©  40 

Press  selected  number.  Then  press  enter. 


(Press  FI  To  Stop) 


(Press  F2  To  Backup) 


1.  Set  1 

2.  Set  2 

3.  Set  3 

4.  Set  4 


Submenu  for  Training  Level  2 

Press  selected  number.  Then  press  enter. 

1.  Level  2:1  multiple  diagrams  of 'all' 

statements 

2.  Level  2:2  multiple  diagrams  of 

'some'  and  'not  air 
statements 


ENTER  A  NUMBER: 


3.  Return  to  main  menu  for  view  of 
training  levels 

- ►  ENTER  A  NUMBER: 


CHARACTERISTIC:  Input  prompts  arc  positioned  in  the  same  area  of  display  consistently. 

DESCRIPTION:  The  input  prompts  are  always  centered  on  the  bottom  of  each  page.  Thus,  this  is  a  signal  to  the  user 
that  a  response  of  some  kind  is  needed.  The  above  examples  illustrate  this  point  with  bold  print 
and  arrows. 


Questions 

1.  Indicate  how  much  this  characteristic  contributed  towards  the  ease  of  interaction  with 
the  computer. 


1  2  3  4  5  6  7 

Low  Medium  High 


2.  Indicate  how  much  this  characteristic  contributed  towards  the  amount  that  was  teamed. 


I  2 
Low 


3  4  5 

Medium 


6  7 

H|gh 
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APPENDIX  B 

MEANS  AND  STANDARD  DEVIATIONS  FOR  EASE  OF  INTERACTION  AND 

TRAINING  GAIN  RATINGS 
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TABLE  B-l.  MEANS  AND  STANDARD  DEVIATIONS  OF  SUBJECTIVE  EVALUATIONS  FOR  EASE  OF 
INTERACTION 


Characteristic 

Mean 

Std.  Dev. 

Brevity 

5.74 

0.95 

Text  broken  into  meaningful  chunks 

5.78 

133 

Seven  to  eight  lines  of  text  per  screen 

5.55 

1.47 

Graphics  take  up  15-25%  of  screen  area 

5.18 

1.78 

Menus  have  no  more  than  5-9  choices 

5.95 

1.18 

Use  of  color,  boxing,  highlighting,  and  text  style  for  important 

5.98 

135 

items 

No  more  than  3-4  screens  without  interactivity 

5.70 

1.32 

Time  required  for  a  session  is  within  attention  span  of  audience 

5.33 

1.47 

Sentences  have  simple  syntax:  active  voice,  not  compounded 

5.93 

1.35 

Data  entries  are  no  more  than  8-10  characters 

6.10 

1.22 

Field  width  for  each  line  is  40  characters  or  less 

5.35 

1.49 

Consistency 

5.79 

0.90 

Functionally  alike  screens  are  formatted  in  the  same  way 

6.15 

1.25 

When  erased,  functional  areas  are  rewritten  in  the  same  order 

6.00 

1.30 

Consistent  use  of  labels  and  graphics 

5.83 

1.24 

Critical  information  comes  at  beginning  of  message  or  centered 

5.98 

1.14 

on  screen 

Constant  delay  of  feedback 

5.58 

1.50 

Similarity  in  the  way  questions  are  asked  and  responses 

6.00 

1.11 

are  made 

Overall  structure  is  clear  through  the  use  of  menus  and  maps 

5.15 

1.59 

A  symbol  always  has  the  same  meaning 

6.08 

1.25 

(table  continues) 
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Characteristic 

Mean 

SttLDev. 

Input  prompts  arc  always  in  the  same  area  of  display 

6.18 

0.98 

Page  number  shown  in  upper-right-hand  corner  for  multi- 

435 

1.98 

screen  transactions 

Flexibility 

5.74 

1.10 

Students  can  page  back  to  review 

5.68 

1.44 

Students  can  exit  lessons,  return  to  menus,  and  exit  the  program 

530 

1.71 

Student  has  control  over  rate  of  presentation  of  frames 

6.35 

1.05 

Student  can  request  more  lengthy  messages  for  further  clarification 

5.53 

135 

Activities  for  diagnosis  of  skills  already  mastered 

5.70 

130 

Remedial  exercises  for  skill  deficiencies 

5.58 

135 

Modularized  program  allows  student  to  begin  at  appropriate  place 

5.7C 

139 

Student  can  choose  difficulty  level  of  problems  or  exercises 

5.53 

132 

Student  can  correct  input  errors 

5.68 

139 

Student  can  choose  an  important  option  and  implement  it  at 

5.70 

131 

any  time 

Compatibility 

5.62 

0.84 

Response  mode  is  appropriate  to  target  audience 

5.50 

1.26 

Students  are  required  to  use  codes  for  responding  only  when 

5.95 

1.06 

necessary 

Visual  information  and  tasks  are  presented  graphically 

5.93 

135 

Where  frames  are  labeled,  title,  not  number,  is  used  for 

4.48 

1.81 

identification 

Input,  output  is  consistent  with  the  user  population  stereotypes 

4.50 

1.80 

Menu  options  are  listed  by  number  where  order  of  lessons 

5.58 

1.34 

is  important 


(tabic  continues) 
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Characteristic 

Mean 

Std.  Dev. 

A  sample  item  is  answered  before  quiz  to  clarify  drill  or  test 

S.68 

1.44 

instructions 

- 

Response  is  demanded  while  instructions  are  on  screen 

6.20 

1.11 

Routing  menus  are  limited  to  3  levels 

5.65 

1.21 

Text  is  displayed  row  by  row 

5.95 

1.24 

Opposite  colors  are  used  to  make  items  distinct 

4.95 

1.84 

Graphics  are  used  far  further  clarification  of  text 

6.08 

1J21 

Menu  selections  are  left  justified  and  in  columns 

5.68 

1.21 

Directions  come  before  menu  selections 

5.98 

1.07 

Responsiveness 

5.70 

0.93 

Periodic  feedback  indicates  normal  operation  when  waiting 

5.95 

1.22 

Computer  tracks  response  patterns  and  gives  option 

5.80 

1.32 

to  pursue  remediation 

Feedback  and  directions  are  distinguishable  bom  other  text 

5.15 

1.66 

At  higher  levels,  more  lengthy  feedback  is  delayed  until 

5.65 

1.46 

end  of  session 

Pause  after  feedback  allows  consolidation  of  material 

5.78 

1.21 

Access  to  helps,  references,  or  resources  are  easily  available 

5.90 

1.10 

Feedback  is  response  specific  at  beginning  of  training 

5.45 

1.24 

Takes  no  more  than  5  seconds  for  text  and  graphics  to  fill  screen 

5.88 

1.20 

Student  gets  more  than  one  chance  to  give  answer 

5.53 

1.57 

L 


B-4 


NTSC  TR87-030 


TABLE  B-2.  MEANS  AND  STANDARD  DEVIATIONS  OF  SUBJECTIVE  EVALUATIONS  FOR  TRAINING  GAIN 


Characteristic 

Mean 

Std.  Dev. 

Brevity 

5.47 

0.86 

Text  broken  into  meaningful  chunks 

5.66 

138 

Seven  to  eight  lines  of  text  per  screen 

5.26 

130 

Graphics  take  up  15-25%  of  screen  area 

5.53 

1.16 

Menus  have  no  more  than  5-9  choices 

5.11 

1.57 

Use  of  color,  boxing,  highlighting,  and  text  style  for  important 

5.61 

134 

items 

No  more  than  3-4  screens  without  interactivity 

6.00 

1.19 

Tune  required  for  a  session  is  within  attention  span  of  audience 

5.03 

1.64 

Sentences  have  simple  syntax:  active  voice,  not  compounded 

6.08 

1.05 

Data  entries  are  no  more  than  8-10  characters 

5.32 

1.45 

Reid  width  for  each  line  is  40  characters  or  less 

5.11 

1.43 

Consistency 

5.14 

0.85 

Functionally  alike  screens  are  formatted  in  the  same  way 

5.95 

1.11 

When  erased,  functional  areas  are  rewritten  in  the  same  order 

5.61 

1.31 

Consistent  use  of  labels  and  graphics 

5.37 

1.36 

Critical  information  comes  at  beginning  of  message  or  centered 

5.32 

1.30 

on  screen 

Constant  delay  of  feedback 

5.39 

1.55 

Similarity  in  the  way  questions  are  asked  and  responses 

5.58 

1.27 

are  made 

Overall  structure  is  clear  through  the  use  of  menus  and  maps 

4.53 

1.59 

A  symbol  always  has  the  same  meaning 

5.34 

1.51 

(tabic  continues) 
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Characteristic 

Mean 

Std.  Dev, 

Input  prompts  are  always  in  the  same  area  of  display 

5.08 

138 

Page  number  shown  in  upper -right-hand  corner  for  multi¬ 

3.24 

1.76 

screen  transactions 

Flexibility 

539 

133 

Students  can  page  back  to  review 

4.74 

2.15 

Students  can  exit  lessons,  return  to  menus,  and  exit  the  program 

432 

233 

Student  has  control  over  rate  of  presentation  of  frames 

639 

1.15 

Student  can  request  more  lengthy  messages  for  further  clarification 

533 

1.87 

Activities  for  diagnosis  of  skills  already  mastered 

6.21 

1.23 

Remedial  exercises  for  skill  deficiencies 

6.03 

135 

Modularized  program  allows  student  to  begin  at  appropriate  place 

4.74 

1.83 

Student  can  choose  difficulty  level  of  problems  or  exercises 

5.42 

1.67 

Student  can  correct  input  errors 

4.58 

2.13 

Student  can  choose  an  important  option  and  implement  it  at 

4.97 

1.79 

any  time 

Compatibility 

5.08 

0.93 

Response  mode  is  appropriate  to  target  audience 

4.95 

1.54 

Students  are  required  to  use  codes  for  responding  only  when 

5.00 

1.51 

necessary 

Visual  information  and  tasks  are  presented  graphical’. 

6.24 

Where  frames  are  labeled,  title,  not  number,  is  used  for 

3.47 

1.59 

identification 

Input,  output  is  consistent  with  the  user  population  stereotypes 

4.00 

2.03 

Menu  options  are  listed  by  number  where  order  of  lessons 

4.50 

1.61 

is  important 

(table  continues) 
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Characteristic 

A  sample  item  is  answered  before  quiz  to  clarify  drill  or  test 
instructions 

Response  is  demanded  while  instructions  sre  on  screen 
Routing  menus  are  limited  to  3  levels 
Text  is  displayed  row  by  row 
Opposite  colors  are  used  to  make  items  distinct 
Graphics  are  used  for  further  clarification  of  text 
Menu  selections  are  left  justified  and  in  columns 
Directions  come  before  menu  selections 
Responsiveness 

Periodic  feedback  indicates  normal  operation  when  waiting 
Computer  tracks  response  patterns  and  gives  option 
to  pursue  remediation 

Feedback  and  directions  are  distinguishable  from  other  text 
At  higher  levels,  more  lengthy  feedback  is  delayed  until 
end  of  session 

Pause  after  feedback  allows  consolidation  of  materal 
Access  to  helps,  references,  or  resources  are  easily  available 
Feedback  is  response  specific  at  beginning  of  training 
Takes  no  more  than  5  seconds  for  text  and  graphics  to  fill  screen 
Student  gets  more  than  one  chance  to  give  answer 


Std.  Dev. 
1.05 


5.84 

1.57 

A 

4.42 

1.48 

A 

5.61 

1.42 

A 

A 

u' 

4,47 

1.90 

■» 

1 

A 

6.63 

0.75 

V 

4.71 

135 

k  r 

a; 

A 

4.84 

1.57 

> 

5.45 

0.87 

3.71 

1.94 

1 

5.97 

1.37 

c 
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